Method and system for anatomical tree structure analysis

ABSTRACT

The present disclosure is directed to a computer-implemented method and system for anatomical tree structure analysis. The method includes receiving model inputs for a set of positions in an anatomical tree structure. The method further includes applying, by a processor, a set of encoders to the model inputs. Each encoder is configured to extract features from the model input at a corresponding position. The method also includes applying, by the processor, a tree structured network to the extracted features. The tree structured network has a plurality of nodes each connected to one or more of the encoders, and information propagates among the nodes of the tree structured network according to spatial constraints of the anatomical tree structure. The method additionally includes providing an output of the tree structured network as an analysis result of the anatomical tree structure analysis.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. application Ser.No. 16/138,946, filed on Sep. 21, 2018, which claims the benefit ofpriority to U.S. Provisional Application No. 62/679,868, filed on Jun.3, 2018, the entire content of which are incorporated herein byreference.

TECHNICAL FIELD

The disclosure generally relates to image processing and analysis. Morespecifically, this disclosure relates to a method and system foranatomical tree structure analysis.

BACKGROUND

Anatomical tree structures commonly exist in human bodies, includinghuman airways, blood vessels (such as arteries, veins, capillaries,etc.), nervous tissues, and breast ducts extending from the nipple, etc.Recent technological advances in medical imaging (CT, MRI, DSA imaging,etc.) make it possible to non-invasively acquire medical images ofdifferent dimensions, such as 2D, 3D, 4D, etc., containing theanatomical tree structure. Clinicians rely on radiologists'interpretation of the medical images to perform various diseasediagnosis, including but not limited to abnormality detection (such aslumen stenosis/widening detection, calcification detection, plaquedetection, etc.), abnormality classification (such as plaque typeclassification among normal, stenosis, widening, calcified plaque,non-calcified plaque and mixed plaque, etc.), parameter quantification(such as abnormality (narrowing, widening, calcification) degreequantification, physiological measurement (diameter, area, flow rate,etc.) estimation, fractional flow reserve estimation), tree branchlabeling (such as labeling the extracted branches with their anatomicalnames), and segmentation (such as vessel lumen segmentation), etc.

Usually, in clinical practice, the anatomical tree structure analysis ismanually performed by a radiologist, which is labor-intensive andtime-consuming, and the results may be subjective. Therefore,automated/semi-automated computer implemented anatomical tree structureanalysis may be adopted to assist the radiologists in improving theefficiency, accuracy, and consistency of the image analysis.

Although machine learning-based algorithms have been introduced for suchsemi-automated or automated image analysis of the anatomical treestructure, these algorithms typically rely on the local features of asingle centerline point or sequential centerline points sampled alongindividual branches, and thus are only able to achieve single point orsequential analysis. More importantly, for the same anatomical treestructure, these algorithms have to analyze the respective branchesasynchronously, which may obtain inconsistent analysis results in thebifurcation regions and overlapped branch regions, reducing the analysisaccuracy and efficiency.

The present disclosure is proposed to address the above concerns.

SUMMARY

The present disclosure intends to provide a method and system foranatomical tree structure analysis, in which a tree structure basedmodel may be generated for a particular task of the anatomical treestructure analysis. The generated model does not consider the featuresof the respective sampling positions in the anatomical tree structureindependently. Instead, it embeds tree structured spatial relationshipsamong the nodes of the recurrent neural network portion (especially theinformation propagation among the nodes) in the model, and takes intoaccount the global dependency of the sampling positions in the wholetree structure. Thus, the generated model may improve the analysisaccuracy and efficiency. Besides, the generated model may obtain theanalysis results for all the sampling positions throughout the branchesin the anatomical tree structure simultaneously, to avoid potentialerrors caused by asynchronous analysis of the branches.

In one aspect, the present disclosure is directed to acomputer-implemented method and system for an anatomical tree structureanalysis. The method may begin with receiving a task of the anatomicaltree structure analysis. Then, a set of positions in the anatomical treestructure may be set, by a processor, as the sampling positions formodel inputs and model outputs. Then a model input may be determined, bythe processor, at each position among the set of positions on the basisof the task. An encoder may be selected, by the processor, for eachposition on the basis of the task. The encoder may be configured toreceive the model input at each position and extract features for thecorresponding sampling position. After that, a tree structured recurrentneural network (RNN) may be constructed by the processor with nodescorresponding to the set of positions and connected with the respectiveencoders. The generated model is therefore adaptive to the task. AN RNNunit for each node may be selected on the basis of the task and aninformation propagation among the nodes may be set on the basis of thespatial constraints of the set of positions in the anatomical treestructure. The tree structured RNN may be provided for performing thetask of the anatomical tree structure analysis.

In another aspect, the present disclosure is directed to acomputer-implemented method for anatomical tree structure analysis. Ananatomical tree structure image acquired by an image acquisition devicemay be received. Then the analysis model for the specific task of theanatomical tree structure analysis may be received. The analysis modelmay be constructed by connecting encoders for a set of positions in theanatomical tree structure with nodes of a tree structured recurrentneural network (RNN). The nodes may correspond to the set of positions.The model input, the encoder, and an RNN unit of each node are selectedbased on the task, and an information propagation among the nodes arebased on the spatial constraints of the set of positions in theanatomical tree structure. The model inputs at the set of positions maybe calculated, by a processor, from the anatomical tree structure image.The specific task of the anatomical tree structure analysis then may beperformed, by the processor, by using the analysis model on the basis ofthe calculated model inputs.

In another aspect, the present disclosure is directed to a system for ananatomical tree structure analysis. The system may include an interfaceconfigured to acquire an anatomical tree structure image, and aprocessor. The processor may be configured to receive a task of theanatomical tree structure analysis. The processor may be furtherconfigured to set a set of positions in the anatomical tree structureand determine a model input at each position among the set of positionson the basis of the task. The processor may be configured to select anencoder for each position on the basis of the task, with the encoderconfigured to receive the model input at each position and extractfeatures for the corresponding position. The processor may be configuredto construct a tree structured RNN with nodes corresponding to the setof positions, by selecting an RNN unit for each node on the basis of thetask and setting an information propagation among the nodes on the basisof the spatial constraints of the set of positions in the anatomicaltree structure. The processor may be configured to connect the nodes ofthe tree structured RNN with the respective encoders. The processor maybe further configured to provide the tree structured RNN for performingthe task of the anatomical tree structure analysis.

In another aspect, the present disclosure is directed to anon-transitory computer readable medium having instructions storedthereon. The instructions, when executed by the processor, may performthe method for an anatomical tree structure analysis as described above.

It is to be understood that the foregoing general description and thefollowing detailed description are exemplary and explanatory only, andare not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingletter suffixes or different letter suffixes may represent differentinstances of similar components. The drawings illustrate generally, byway of example, but not by way of limitation, various embodiments, andtogether with the description and claims, serve to explain the disclosedembodiments. When appropriate, the same reference numbers are usedthroughout the drawings to refer to the same or like parts. Suchembodiments are demonstrative and not intended to be exhaustive orexclusive embodiments of the present method, device, or non-transitorycomputer readable medium having instructions thereon for implementingthe method.

FIG. 1 illustrates an exemplary computer-implemented process foranatomical tree structure analysis, according to an embodiment ofpresent disclosure;

FIG. 2 illustrates an exemplary anatomical tree structure analysissystem, according to an embodiment of present disclosure;

FIG. 3 illustrates an exemplary tree structure based learning model,according to an embodiment of present disclosure;

FIG. 4 illustrates an exemplary vessel stenosis degree predictionprocess, according to an embodiment of present disclosure;

FIG. 5 illustrates an exemplary tree structure based learning model,according to an embodiment of present disclosure;

FIG. 6 illustrates an exemplary tree structure based learning model,according to an embodiment of present disclosure;

FIG. 7 illustrates an exemplary vessel stenosis segmentation processusing a segmentation model, according to an embodiment of presentdisclosure;

FIG. 8 illustrates an exemplary tree structure based learning modeltraining process, according to an embodiment of present disclosure; and

FIG. 9 depicts a block diagram illustrating an exemplary anatomical treestructure analyzing system, according to an embodiment of presentdisclosure.

DETAILED DESCRIPTION

Consistent with the present disclosure, the technical term “treestructure” may refer to one or more branches. The technical term“branch” refers to one of the physiological tubes (e.g. vessel tubes)stemming from a bifurcation point. The technical term “path” refers to apass from the inlet to an outlet of the anatomical tree structure.

FIG. 1 illustrates an exemplary computer-implemented method 100 foranatomical tree structure analysis, according to an embodiment ofpresent disclosure. As shown in FIG. 1, the method 100 begins withreceiving a task of anatomical tree structure analysis (step 101). Insome embodiments, the task of the anatomical tree structure analysis maybe assigned by a user by selecting the analysis task options in a menuof the image analysis software. For example, when the user checks the“FFR prediction” option, the “FFR prediction” task is received at step101. In some embodiments, the task of anatomical tree structure analysismay be customized for the user with the image analysis software. As anexample, as a user request, a vessel plaque segmentation task may becustomized, and the customized vessel plaque segmentation task may bereceived at step 101. In some embodiments, the method 100 may beimplemented by the image analysis software to perform adaptive imageanalysis functions. Unlike the current image analysis software with afixed framework of image analysis model, the image analysis softwareimplementing the method 100 allows adaptive modeling for variableanalysis tasks.

In step 102, a set of positions in the anatomical tree structure may beset or received, by a processor, as the sampling positions for modelinputs and model outputs. In some embodiments, the set of positions maybe set automatically by a processor. As an example, upon receiving thevessel tree structure image, the processor may perform vessel wall andcenterline (as an example of the skeleton line) extraction and set thesampling positions along the centerline. As an example, the bifurcationpoints of the centerline may be included, which usually carry anatomicalmeaningful information and assist the tree structured RNN in accuratelyaccounting for the global dependency of the positions in the whole tree.In some embodiments, the processor may extract the branches and thebifurcation points in the vessel tree structure and set at least onepoint in each branch in addition to the bifurcation points as the set ofsampling positions. In this manner, the tree structured RNN mayaccurately and completely take into account the global dependency of thepositions in the whole tree.

In some embodiments, the set of positions may be set semi-automaticallyby the processor. As an example, the user (e.g. a radiologist,physician, clinician, etc.) may manually assign the number of the pointsin each branch besides the bifurcation points or assign the analysisresolution (e.g., 0.2 mm), and the processor may set the samplingpositions accordingly.

In some embodiments, the set of positions may be set manually by theuser. As an example, an anatomical tree structure image may be acquiredand presented to the user for him/her to manually set the samplingpositions. As another example, the skeleton line of the anatomical treestructure may be extracted and present to the user for him/her tomanually set the sampling positions along the skeleton line. In thismanner, the user may incorporate the points of interest, such as thecandidate stenosis, the bifurcation points, etc., into the samplingpositions, to ensure that the model may obtain the analysis results atthe sampling positions as needed by the diagnosis.

In step 103, model inputs may be selected, by the processor at thesampling positions on the basis of the task. For neural network-basedanalyzing model, various model inputs may be adopted, including but notlimited to features (geometrical features, flow features, etc.) or imagepatches along the skeleton line of the anatomical tree structure. Instep 103, proper type of model inputs may be selected on the basis ofthe particular task. As an example, under the condition that the task isany one of abnormality detection (e.g., disease detection), abnormalityclassification (e.g., disease labeling), parameter quantification (e.g.,to quantify continuous measurements associated with the anatomical treestructure), or labeling (labeling the extracted branches with theiranatomical names). Image patches or feature vectors may be selected asthe model inputs. As another example, under the condition that the taskis segmentation (e.g., tumor segmentation, stenosis segmentation, etc.),image patches may be adopted as the model inputs.

In step 104, an encoder may be selected, by the processor, for eachposition of the set of positions on the basis of the task. The encodermay be configured to receive the model inputs at each position andextract features for the corresponding position. The encoder may be usedto extract features for the model input at the corresponding samplingposition, from which local-relevant information may be extracted. Thefeatures may form a feature vector for abnormality detection,abnormality classification, or parameter quantification tasks, and/ormay be a feature map for segmentation task. In contrast to using fixedfeature as model inputs, the disclosed encoder may encode hidden featureinformation, especially the higher-level feature information. In someembodiments, to perform tasks such as abnormality detection, abnormalityclassification, or parameter quantification, at least one ofconvolutional neural network (CNN), fully convolutional neural network(FCN), and multi-layer perceptron (MLP) may be selected as the encoder.In some embodiments, under the condition that the task is segmentation,CNN or FCN may be selected as the encoder.

After that, in step 105, a tree structured recurrent neural network(RNN) may be constructed by the processor with nodes corresponding tothe set of positions. In some embodiments, proper RNN units may beselected for each node on the basis of the task. As an example, toperform tasks such as abnormality detection, abnormality classification,or parameter quantification, long short-term memory (LSTM) or gaterecurrent unit (GRU) may be selected as an RNN unit. As another example,to perform a segmentation task, convolutional LSTM (CLSTM) orconvolutional GRU (CGRU) may be selected as the RNN unit.

In some embodiments, the RNN model is designed to include an encoder totransform each model input to produce its feature vector/maprepresentation, which will be passed onto the tree structured RNN model.As a result, the RNN model is adaptive for the task. In someembodiments, the RNN unit for each node may be selected on the basis ofthe task and the information propagation among the nodes may be set onthe basis of the spatial constraints of the set of positions in theanatomical tree structure. With the information propagation among thenodes, the information from the sampling positions in the whole tree maybe seamlessly integrated to improve accuracy of the image analysis. Inaddition, the analysis results for all the sampling positions may beobtained simultaneously, which further improves the analysis accuracyand efficiency by avoiding the additional time consumption as well aspotential errors and inconsistencies caused by asynchronous processingof different positions/branches.

In some embodiments, the information propagation among the nodes of thetree structured RNN may be set to conform to the spatial constraints ofthe set of sampling positions in the anatomical tree structure. As anexample, if two sampling positions are spatially connected in a vesselbranch, the corresponding two nodes are connected. As another example,if a first sampling position and a second sampling position are locatedin two respective vessel branches and are connected with each other viaa third sampling position at the bifurcation point, the third nodecorresponding to the third sampling position connects the first nodecorresponding to the first sampling position with the second nodecorresponding to the second sampling position. In some embodiments,bidirectional information propagation may be allowed between each pairof nodes corresponding to two adjacent positions in a path of theanatomical tree structure. Alternatively, unidirectional informationpropagation from distal side to root may be set between at least onepair of nodes corresponding to two adjacent positions in the path of theanatomical tree structure. In this manner, the node tree structuremaintains the topology of the anatomical tree structure and simulatesaccurately the global physical acting mechanism of the points in theanatomical tree structure, which improves the analysis accuracy andefficiency of the analysis model.

In step 106, the analysis model generated adaptively may be used toimplement variable anatomical tree structure analysis tasks.Specifically, an anatomical tree structure image may be acquired. Asdescribed above, the analysis model may be constructed by connectingencoders for a set of positions in the anatomical tree structure withthe corresponding nodes of a tree structured recurrent neural network(RNN) with nodes corresponding to the set of positions, wherein modelinput, encoder, and RNN unit of each node are based on the task, andinformation propagation among the nodes are based on the spatialconstraints of the set of positions in the anatomical tree structure. Toapply the analysis model, model inputs at the set of positions may becalculated from the anatomical tree structure image acquired for thespecific task of the anatomical tree structure analysis. The calculatedmodel inputs are then input into the analysis model.

FIG. 2 illustrates an exemplary anatomical tree structure analysissystem 200 according to an embodiment of present disclosure. As shown inFIG. 2, the anatomical tree structure analysis system 200 may include ananalysis model generation unit 202, an analysis model training unit 203,and an analysis unit 204. The analysis model generation unit 202 may beconfigured to receive task option, sampling position option, and modelinput option, select and retrieve proper encoders and RNN units from theneural network library 201 based on the received options, and generatethe analysis model with the retrieved encoders and RNN units. In someembodiments, analysis model may be comprised of an encoder connectedwith each node of the tree structured RNN model where the RNN nodescorresponding to the sampling positions. The retrieved encoders may beused for each node of the encoder portion, the retrieved RNN units maybe used for each node of the tree structured RNN portion, andinformation propagation among the nodes of the tree structured RNNportion may be set on the basis of the spatial constraints of thesampling positions in the anatomical tree structure.

In some embodiments, the analysis model may be transmitted from theanalysis model generation unit 202 to the analysis model training unit203 to be trained. In some embodiments, the analysis model training unit203 may obtain corresponding training samples from the training sampledatabase 204 on the basis of the task option, sampling position option,and model input option and train the analysis model with the obtainedtraining samples. As an example, for the task option as “vessel stenosislabel prediction,” the sampling positions option as “centerline pointsat regular interval,” and the model input option as “vessel diameter,”vessel images annotated with vessel diameter and stenosis labels may beobtained from the training sample database 204 as training samples totrain the analysis model.

In some embodiments, the trained analysis model may be transmitted fromthe analysis model training unit 203 to the analysis unit 204. Theanalysis unit 204 may receive model inputs from a model input extractionunit 205 and perform the analysis using the trained analysis model onthe basis of the received model inputs. The model input extraction unit205 may receive the sampling position option and model input option andextract the model inputs from medical images it receives from themedical image database 206 on the basis of the received samplingposition options and model input options. As an example, for thesampling position option as “centerline points at regular interval” andthe model input option as “vessel diameter,” the model input extractionunit 205 may obtain vessel angiography images at different projectionangles from the medical image database 206, reconstruct a 3D vesselmodel from the obtained vessel angiography images, and extract thevessel diameters at centerline points at the regular interval as themodel inputs.

FIG. 3 illustrates an exemplary tree structure based learning modelaccording to an embodiment of present disclosure. As shown in FIG. 3,the anatomical tree structure may be a vessel tree, and the model inputsx may be image patches or feature vectors at centerline points (samplingpoints) of the vessel tree. As an example, artificially defined featurevectors at centerline points of the vessel tree may be adopted as themodel inputs x. As an example, the model inputs x may be vesseldiameters at centerline sampling points of the vessel tree. The learningmodel is a combination of two portions to form an end-to-end network.One portion is the encoder portion and the other portion is a treestructured RNN. In some embodiments, the encoder is configured toconstruct a feature vector (or feature map) for each model input x, fromwhich local-relevant higher-level feature information may be extracted.The tree structure RNN portion may be comprised of nodes correspondingto the centerline sampling points, with bi-directional RNN as the node.Feature information (such as feature vector or feature map) may betransmitted from each encoder to the corresponding node of the treestructure RNN portion. Bidirectional information propagation may be setbetween each pair of nodes corresponding to two adjacent centerlinesampling points in a path of the vessel tree, to handle the spatialconstraint in the vessel tree. The nodes of the tree structure RNNportion may integrate the feature information at all the centerlinesampling points in the whole vessel tree, to make predictions(prediction result y) for all centerline sampling points in the vesseltree simultaneously. It may avoid potential errors caused bypre-processing or post-processing. As shown in FIG. 3, two bifurcationpoints along the centerline are incorporated into the sampling points.As an example, each branch of the vessel tree may include at least onesampling point other than the bifurcation points.

FIG. 4 illustrates an exemplary vessel stenosis degree predictionprocess 400 according to an embodiment of present disclosure. Theprediction process 400 begins with centerline extraction 401. In someembodiments, vessel images may be acquired using imaging devices, andcenterline may be extracted from the acquired vessel images. Then, imagepatches or features may be extracted on the centerline (step 402) as theinputs of the prediction model comprised of encoders and bi-tree-RNN. Ifgeometrical features along the centerline are used as the model inputs,centerline and vessel wall may be extracted from the acquired vesselimages to reconstruct a 3D vessel model and extract the geometricalfeatures along the centerline. The model inputs may be fed into thetrained prediction model (step 403), to output prediction results of thevessel stenosis degree along the centerline.

In some embodiments, an exemplary tree structure based learning model asshown in FIG. 5 may be used by the vessel stenosis degree predictionprocess 400. As shown in FIG. 5, x may be image patches or featurevectors, y may be continuous parameter values (e.g. vessel stenosisdegree) along the vessel centerline, the encoder may be a CNN and/or anMLP, and the RNN may be a BLSTM and/or a GRU.

FIG. 6 illustrates an exemplary tree structure based learning modelaccording to an embodiment of present disclosure. As shown in FIG. 6, xmay be image patches, y may be segmentation masks, the encoder may be aCNN and/or an FCN, and the RNN may be a CLSTM and/or a CGRU. Thelearning model as shown in FIG. 6 may be used as segmentation model forimage segmentation task, e.g. vessel stenosis segmentation process.

FIG. 7 illustrates an exemplary vessel stenosis segmentation process 700using a segmentation model generated by the method according to anembodiment of present disclosure. The vessel stenosis segmentationprocess 700 may be performed in an online manner, and may begin withreceiving vessel tree images, which may be acquired and transmitted byvarious imaging devices (step 701). In step 702, a centerline may beextracted from the received vessel tree images and image patches may beextracted at sampling positions along the centerline as the modelinputs. Then the image patches at the sampling positions along thecenterline may be fed (one-to-one) into the encoders of the trainedsegmentation model, to calculate the segmentation masks at the samplingpositions along the centerline as the model outputs (step 703). Afterthat, the calculated segmentation masks may be transformed into acoordinate system of the vessel tree image and mapped onto the vesseltree (step 704), to present the segmentation results in an intuitivemanner to the user. By means of the vessel stenosis segmentation process700, the vessel stenosis segmentation may be performed from end to endautomatically. That is, the user may input the acquired vessel treeimages and receive the vessel stenosis segmentation result directly,which may be presented on a display.

In some embodiments, the analysis model as described above may betrained in an off-line manner. FIG. 8 illustrates an exemplary analysismodel training process 800, according to an embodiment of presentdisclosure. The process 800 may begin with receiving the trainingdataset (step 801). The received training dataset may be divided intobatches (step 802), which may be individually loaded as a currenttraining data (step 803). For example, the inputs of the analysis modelmay be denoted as X={x₁, x₂, . . . , x_(t)} and the outputs of theanalysis model may be denoted as Y={y₁, y₂, . . . , y_(t)},respectively. For different image analysis tasks, inputs X may be imagepatches or feature vectors extracted from the anatomical tree structureat centerline points, and Y may be disease class labels, continuousmeasurements, segmentation mask, etc.

The parameters of the analysis model may be determined based on thebatch of training data (step 804) and validated against a loss function(step 805) to be optimized for the batch of training data. As describedabove, the analysis model may be constructed by connecting an encoderwith a corresponding node of a tree structured RNN. The analysis modelthus may contain parameters (V, W) with parameters V for the encoderportion and parameters W for the tree structured RNN. In someembodiments, the parameters (V, W) may be jointly optimized byminimizing a loss function. As an example, the loss function may be themean square error of the ground truth outputs 9 and the model outputvalues y_(t) at each position t within the batch. In some embodiments,the analysis model may be trained using gradient descent related methodsto optimize the loss function with respect to all parameters (V, W) overeach batch. As an example, for each batch, the mean square error may becalculated for each training sample in the batch, and gradient may becalculated based on the same and averaged. The analysis model,especially its parameters, may be updated based on the averagedgradient. Although gradient descent related methods and mean squareerror are disclosed as examples, other functions may be adoptedincluding but not limited to cross entropy, etc., and other parameteroptimizing methods may also be adopted, including but not limited toadaptive moment estimation, etc. Upon confirmation that all batches areprocessed in step 806, the analysis model, whose parameter have beenoptimized over all the batches, may be output (step 807).

The process 800 may adopt a mini-batch gradient descent method as anexample. As an alternative, it may also adopt gradient descent orstochastic gradient descent methods. The mini-batch gradient descentmethod may achieve more robust convergence meanwhile efficientlyavoiding local optimization with a relatively high computing efficiency.Besides, the memory does not need to load large amounts of trainingdataset for medical image analysis as a whole. Instead, the trainingsamples may be loaded in batches, which relieves the working load of thememory and improves its working efficiency.

FIG. 9 illustrates a block diagram of an exemplary anatomical treestructure analyzing system 900 according to an embodiment of presentdisclosure. The anatomical tree structure analyzing system 900 mayinclude a network interface 908, by means of which the anatomical treestructure analyzing system 900 may be connected to the network (notshown), such as but not limited to the local area network in thehospital or the Internet. The network can connect the anatomical treestructure analyzing system 900 with external devices such as an imageacquisition device (not shown), medical image database 905, and an imagedata storage device 906. An image acquisition device may be any type ofimaging modalities, such as but not limited to CT, digital subtractionangiography (DSA), MRI, functional MRI, dynamic contrast enhanced MRI,diffusion MRI, spiral CT, cone beam computed tomography (CBCT), positronemission tomography (PET), single-photon emission computed tomography(SPECT), X-ray, optical tomography, fluorescence imaging, ultrasoundimaging, radiotherapy portal imaging.

In some embodiments, the anatomical tree structure analyzing device 900may be a dedicated intelligent device or a general purpose intelligentdevice. For example, the device 900 may be a computer customized forimage data acquisition and image data processing tasks, or a serverplaced in the cloud. For example, the device 900 may be integrated intothe image acquisition device. Optionally, the image processingprogram(s) 903 in the device 900 may include or cooperate with a 3Dreconstruction unit for reconstructing the 3D model of the vessel on thebasis of the 2D vessel images acquired by the image acquisition device,and extract geometrical features from the 3D model at a set ofcenterline points as image analysis model inputs X={x₁, x₂, . . . ,x_(t)}.

The anatomical tree structure analyzing system 900 may include an imageprocessor 901 and a memory 902, and may additionally include at leastone of an input/output 907 and an image display 909.

The image processor 901 may be a processing device that includes one ormore processing devices, such as a microprocessor, a central processingunit (CPU), a graphics processing unit (GPU), and the like. Morespecifically, the image processor 901 may be a complex instruction setcomputing (CISC) microprocessor, a reduced instruction set computing(RISC) microprocessor, a very long instruction word (VLIW)microprocessor, a processor running other instruction sets, or aprocessor that runs a combination of instruction sets. The imageprocessor 901 may also be one or more dedicated processing devices suchas application specific integrated circuits (ASICs), field programmablegate arrays (FPGAs), digital signal processors (DSPs), system-on-chip(SoCs), and the like. As would be appreciated by those skilled in theart, in some embodiments, the image processor 901 may be aspecial-purpose processor, rather than a general-purpose processor. Theimage processor 901 may include one or more known processing devices,such as a microprocessor from the Pentium™, Core™, Xeon™, or Itanium®family manufactured by Intel™, the Turion™, Athlon™, Sempron™, Opteron™FX™, Phenom™ family manufactured by AMD™, or any of various processorsmanufactured by Sun Microsystems. The image processor 901 may alsoinclude graphical processing units such as a GPU from the GeForce®,Quadro®, Tesle® family manufactured by Nvidia™, GMA, Iris™ familymanufactured by Intel™, or the Radeon™ family manufactured by AMD™. Theimage processor 901 may also include accelerated processing units suchas the Desktop A-4 (9, 9) Series manufactured by AMD™, the Xeon Phi™family manufactured by Intel™. The disclosed embodiments are not limitedto any type of processor(s) or processor circuits otherwise configuredto meet the computing demands of identifying, analyzing, maintaining,generating, and/or providing large amounts of imaging data ormanipulating such imaging data to, or to manipulate any other type ofdata consistent with the disclosed embodiments. In addition, the term“processor” or “image processor” may include more than one processor,for example, a multi-core design or a plurality of processors eachhaving a multi-core design. The image processor 901 can executesequences of computer program instructions, stored in memory 902, toperform various operations, processes, methods disclosed herein.

The image processor 901 may be communicatively coupled to the memory 902and configured to execute computer-executable instructions storedtherein. The memory 902 may include a read only memory (ROM), a flashmemory, random access memory (RAM), a dynamic random-access memory(DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM, a static memory(e.g., flash memory, static random access memory), etc., on whichcomputer executable instructions are stored in any format. In someembodiments, the memory 902 may store computer-executable instructionsof one or more image processing program(s) 903. The computer programinstructions can be accessed by the image processor 901, read from theROM, or any other suitable memory location, and loaded in the RAM forexecution by the image processor 901. For example, memory 902 may storeone or more software applications. Software applications stored in thememory 902 may include, for example, an operating system (not shown) forcommon computer systems as well as for soft-controlled devices.

Further, memory 902 may store an entire software application or only apart of a software application (e.g. the image processing program (s)903) to be executable by the image processor 901. In addition, thememory 902 may store a plurality of software modules, for implementingthe respective steps of the method for anatomical tree structureanalysis consistent with the present disclosure. For example, theanalysis model generation unit 202, the analysis model training unit203, the analysis unit 204, and the model input extraction unit 205 (asshown in FIG. 2), may be implemented as soft modules stored on thememory 902. For another example, at least the analysis unit 204 and themodel input extraction unit 205 may be implemented as soft modulesstored on the memory 902, the analysis model generation unit 202 andanalysis model training unit 203 may be located remote from theanatomical tree structure analyzing system 900 and communicate with theanalysis unit 204 to enable it receive the updated analysis model, whichmay be generated by the analysis model generation unit 202 on the basisof the particular task and trained by the analysis model training unit203 with the training sample from the training sample database 204 (inan off-line training process) and/or from the analysis unit 204 (i.e.,the analysis results automatically or semi-automatically generatedtherefrom (with or without manual correction by the user) together withthe corresponding extracted model inputs) (in an on-line trainingprocess).

Besides, the memory 902 may store data generated/buffered when acomputer program is executed, for example, medical image data 904,including the medical images transmitted from image acquisitiondevice(s), medical image database 905, image data storage device 906,etc. In some embodiments, medical image data 904 may include thereceived image(s) of the vessel tree, for which centerline extractionand 3D model reconstruction, the automatic geometrical featureextraction (as model inputs) and further image analysis (e.g., vesselstenosis degree prediction results) are to be implemented by the imageprocessing program(s) 903. In some embodiment, medical image data 904may include the received volumetric image of the vessel tree, for whichthe automatic geometrical feature extraction (as model inputs) andfurther image analysis (e.g., vessel stenosis degree prediction results)are to be implemented by the image processing program(s) 903. In someembodiments, the memory 902 may load a batch of training samples fromthe medical image database 905 and temporarily store the same as medicalimage data 904, to be utilized by the analysis model training unit 203for mini-batch training. In some embodiments, the memory 902 may storetemporarily the automatic image analysis results associated with theactual model inputs as on-line training samples. The training samplesstored as the medical image data 904 may be cancelled after the trainingutilizing the same is complete, to release the space of the memory 902and improve its capacity and performance.

In some embodiments, the generated analysis model for a task may bestored in the medical image data 904 and may be used (after trained) inthe next image analysis of the same task. In some embodiments, theupdated and optimized parameters of the trained analysis model may bestored in the medical image data 904, which may then be used in the nextimage analysis of the same task on the same patient. As an example,confronting a task of a coronary vessel stenosis degree prediction, theimage processor 901 may retrieve the corresponding prediction modelalready generated and/or trained from the medical image data 904 andmake use of the same (e.g. use it upon transforming training based onnew training samples). As another example, confronting a task of acoronary vessel stenosis degree prediction on the same patient, theimage processor 901 may retrieve the prediction model latest updated forthe same patient from the medical image data 904 and use it directly.

In some embodiments, the image processor 901, upon performing an imageanalysis task, may associate the images of the tree structure togetherwith the analysis results as medical image data 904 for presentingand/or transmitting. In some embodiments, the tree structure imagestogether with the analysis results, e.g., the vessel tree images andlumen refine segmentation results, may be displayed on the image display909 for the user's review. For example, the image display 909 may be anLCD, a CRT, or an LED display. In this manner, the user may confirm andcorrect the displayed analysis results by means of the input/output 907,if necessary. And the confirmed and corrected image analysis results maybe temporarily stored associated with the model inputs as medical imagedata 904 in the memory 902 and may be transmitted to the medical imagedatabase 905, to be accessed, obtained, and utilized by other medicaldevices (e.g. other anatomical tree structure analyzing devices 900), ifneeded.

In some embodiments, the memory 902 may communicate with the medicalimage database 905 to transmit and save the extracted model inputsassociated with the automatically or semi-automatically obtainedanalysis results into the same as a piece of training data, which may beused for off-line training. In this manner, the training sample database204 as shown in FIG. 2 may be incorporated into the medical imagedatabase 905.

Besides, the parameters of the generated and/or trained analysis modelmay be stored in the medical image database 905, to be accessed,obtained, and utilized by other anatomical tree structure analyzingdevices 900, if needed. In some embodiments, the neural network library201 as shown in FIG. 2 may be included in the medical image database905, so that the analysis model generation unit 202, if included as theimage processing program(s) 903, may retrieve encoders and/or treestructured RNNs from the neural network library 201 in the medical imagedatabase 905.

In some embodiments, the medical image database 206 as shown in FIG. 2may be incorporated into the medical image database 905, which maymaintain medical images and/or 3D models and/or skeleton lines of theanatomical tree structures in accordance with patients. Thereby, thememory 902 may communicate with the medical image database 905 to obtainthe images and/or 3D models and/or skeleton lines of the anatomical treestructure of the current patient.

In some embodiments, the image data storage device 906 may be providedto exchange image data with the medical image database 905. For example,the image data storage device 906 may reside in other medical imageacquisition devices, e.g., a CT which performs volumetric scan on thepatients. The volumetric images of the patients may be transmitted andsaved into the medical image database 905, and the anatomical treestructure analyzing device 900 may retrieve volumetric images andanalysis models of a specific patient from the medical image database905 and make image analysis on the basis of the same.

The input/output 907 may be configured to allow the anatomical treestructure analyzing device 900 to receive and/or send data. Theinput/output 907 may include one or more digital and/or analogcommunication devices that allow the device 900 to communicate with auser or other machine and device. For example, the input/output 907 mayinclude a keyboard and a mouse that allow the user to provide an input,including but not limited to task option, sampling position option,model input option, etc., as shown in FIG. 2.

The network interface 908 may include a network adapter, a cableconnector, a serial connector, a USB connector, a parallel connector, ahigh-speed data transmission adapter such as optical fiber, USB 9.0,lightning, a wireless network adapter such as a Wi-Fi adapter, atelecommunication (9G, 4G/LTE, etc.) adapters. The device 900 may beconnected to the network through the network interface 908. The networkmay provide the functionality of local area network (LAN), a wirelessnetwork, a cloud computing environment (e.g., software as a service,platform as a service, infrastructure as a service, etc.), aclient-server, a wide area network (WAN), and the like.

Various operations or functions are described herein, which may beimplemented as software code or instructions or defined as software codeor instructions. Such content may be source code or differential code(“delta” or “patch” code) that can be executed directly (“object” or“executable” form). The software code or instructions may be stored incomputer readable storage medium, and when executed, may cause a machineto perform the described functions or operations and include anymechanism for storing information in the form accessible by a machine(e.g., computing device, electronic system, etc.), such as recordable ornon-recordable media (e.g., read-only memory (ROM), random access memory(RAM), disk storage media, optical storage media, flash memory devices,etc.).

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations of theembodiments will be apparent from consideration of the specification andpractice of the disclosed embodiments.

In this document, the terms “a”, “an”, and “the” are used, as is commonin patent documents, to include one or more than one, independent of anyother instances or usages of “at least one” or “one or more.” Thus, forexample, reference to “a level” includes a plurality of such levels, andso forth.

In this document, the term “or” is used to refer to a nonexclusive or,such that “A or B” includes “A but not B,” “B but not A,” and “A and B,”unless otherwise indicated. In this document, the terms “including” and“in which” are used as the plain-English equivalents of the respectiveterms “comprising” and “wherein.” Also, in the following claims, theterms “including” and “comprising” are open-ended. That is, the term“comprising”, which is synonymous with “including” “containing” or“characterized by” is inclusive or open-ended and does not excludeadditional, unrecited elements or method steps. “Comprising” is a termof art used in claim language which means that the named elements areessential, but other elements can be added and still form a constructwithin the scope of the claim. An apparatus, system, device, article,composition, formulation, or process that includes elements in additionto those listed after such a term in a claim are still deemed to fallwithin the scope of that claim. Moreover, in the following claims, theterms “first,” “second,” and “third,” etc. are used merely as labels,and are not intended to impose numerical requirements on their objects.

Exemplary Methods described herein can be machine orcomputer-implemented at least in part. Some examples can include acomputer-readable medium or machine-readable medium encoded withinstructions operable to configure an electronic device to performmethods as described in the above examples. An implementation of suchmethods can include software code, such as microcode, assembly languagecode, a higher-level language code, or the like. The various programs orprogram modules can be created using a variety of software programmingtechniques. For example, program sections or program modules can bedesigned in or by means of Java, Python, C, C++, assembly language, orany known programming languages. One or more of such software sectionsor modules can be integrated into a computer system and/orcomputer-readable media. Such software code can include computerreadable instructions for performing various methods. The software codemay form portions of computer program products or computer programmodules. Further, in an example, the software code can be tangiblystored on one or more volatile, non-transitory, or non-volatile tangiblecomputer- readable media, such as during execution or at other times.Examples of these tangible computer-readable media can include, but arenot limited to, hard disks, removable magnetic disks, removable opticaldisks (e.g., compact disks and digital video disks), magnetic cassettes,memory cards or sticks, random access memories (RAMs), read onlymemories (ROMs), and the like.

As used herein, the term “and/or” when used in the context of a listingof entities, refers to the entities being present singly or incombination. Thus, for example, the phrase “A, B, C, and/or D” includesA, B, C, and D individually, but also includes any and all combinationsand sub combinations of A, B, C, and D.

Moreover, while illustrative embodiments have been described herein, thescope includes any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations or alterations based on the presentdisclosure. The elements in the claims are to be interpreted broadlybased on the language employed in the claims and not limited to examplesdescribed in the present specification or during the prosecution of theapplication, which examples are to be construed as non-exclusive.Further, the steps of the disclosed methods can be modified in anymanner, including by reordering steps or inserting or deleting steps. Itis intended, therefore, that the descriptions be considered as examplesonly, with a true scope being indicated by the following claims andtheir full scope of equivalents.

The above description is intended to be illustrative, and notrestrictive. For example, the above-described examples (or one or moreaspects thereof) may be used in combination with each other. Otherembodiments can be used, such as by one of ordinary skill in the artupon reviewing the above description. Also, in the above DetailedDescription, various features may be grouped together to streamline thedisclosure. This should not be interpreted as intending that anunclaimed disclosed feature is essential to any claim. Rather, inventivesubject matter may lie in less than all features of a disclosedembodiment. Thus, the following claims are hereby incorporated into theDetailed Description as examples or embodiments, with each claimstanding on its own as a separate embodiment, and it is contemplatedthat such embodiments can be combined with each other in variouscombinations or permutations. The scope of the invention should bedetermined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled.

What is claimed is:
 1. A computer-implemented method for an anatomicaltree structure analysis, comprising: receiving model inputs for a set ofpositions in an anatomical tree structure, wherein the anatomical treestructure includes at least one bifurcation point and a plurality ofbranches splitting from the at least one bifurcation point; applying, bya processor, a set of encoders to the model inputs, wherein each encoderis configured to extract features from the model input at acorresponding position; applying, by the processor, a tree structurednetwork to the extracted features, wherein the tree structured networkhas a plurality of nodes each connected to one or more of the encoders,wherein information propagates among the nodes of the tree structurednetwork according to spatial constraints of the anatomical treestructure; and providing an output of the tree structured network as ananalysis result of the anatomical tree structure analysis.
 2. The methodof claim 1, wherein the anatomical tree structure is a blood vessel oran airway.
 3. The method of claim 1, wherein the set of positionsinclude the at least one bifurcation point and at least one point ineach branch.
 4. The method of claim 1, further comprising: receiving animage of the anatomical tree structure acquired by an image acquisitiondevice; and deriving the model inputs at the set of positions in theanatomical tree structure from the image.
 5. The method of claim 1,wherein the encoders are selected from a convolutional neural network(CNN), a fully convolutional neural network (FCN), and a multi-layerperceptron (MLP).
 6. The method of claim 1, wherein the tree structurednetwork is a recurrent neural network (RNN) that includes a plurality ofRNN unit each corresponding to a node.
 7. The method claim 6, whereinthe RNN units are selected from a long short-term memory (LSTM) and agate recurrent unit (GRU).
 8. The method of claim 1, wherein theinformation propagates bi-directionally between a pair of nodescorresponding to two adjacent positions in a path of the anatomical treestructure.
 9. The method of claim 1, wherein the information propagatesin a single direction between a pair of nodes, wherein the singledirection is either from a distal side to a root between the pair ofnodes, or from the root to the distal side between the pair of nodes.10. The method of claim 1, wherein the set of encoders and the treestructured network are trained jointly.
 11. A system for performing ananatomical tree structure analysis, comprising: an interface, configuredto receive model inputs for a set of positions in an anatomical treestructure, wherein the anatomical tree structure includes at least onebifurcation point and a plurality of branches splitting from the atleast one bifurcation point; and a processor, configured to: apply a setof encoders to the model inputs, wherein each encoder is configured toextract features from the model input at a corresponding position; applya tree structured network to the extracted features, wherein the treestructured network has a plurality of nodes each connected to one ormore of the encoders, wherein information propagates among the nodes ofthe tree structured network according to spatial constraints of theanatomical tree structure; and provide an output of the tree structurednetwork as an analysis result of the anatomical tree structure analysis.12. The system of claim 11, wherein the anatomical tree structure is ablood vessel or an airway.
 13. The system of claim 11, wherein the setof positions include the at least one bifurcation point and at least onepoint in each branch.
 14. The system of claim 11, wherein the interfaceis further configured to receive an image of the anatomical treestructure acquired by an image acquisition device, and wherein theprocessor is further configured to derive the model inputs at the set ofpositions in the anatomical tree structure from the image.
 15. Thesystem of claim 11, wherein the encoders are selected from aconvolutional neural network (CNN), a fully convolutional neural network(FCN), and a multi-layer perceptron (MLP).
 16. The system of claim 11,wherein the tree structured network is a recurrent neural network (RNN)that includes a plurality of RNN unit each corresponding to a node. 17.The system of claim 16, wherein the RNN units are selected from a longshort-term memory (LSTM) and a gate recurrent unit (GRU).
 17. The systemof claim 11, wherein the information propagates bi-directionally betweena pair of nodes corresponding to two adjacent positions in a path of theanatomical tree structure.
 18. The system of claim 11, wherein theinformation propagates in a single direction between a pair of nodes,wherein the single direction is from a distal side to a root between thepair of nodes, or from the root to the distal side between the pair ofnodes.
 19. A non-transitory computer readable medium having instructionsstored thereon, the instructions, when executed by a processor, performa method for an anatomical tree structure analysis, the methodcomprising: receiving model inputs for a set of positions in ananatomical tree structure, wherein the anatomical tree structureincludes at least one bifurcation point and a plurality of branchessplitting from the at least one bifurcation point; applying a set ofencoders to the model inputs, wherein each encoder is configured toextract features from the model input at a corresponding position;applying a tree structured network to the extracted features, whereinthe tree structured network has a plurality of nodes each connected toone or more of the encoders, wherein information propagates among thenodes of the tree structured network according to spatial constraints ofthe anatomical tree structure; and providing an output of the treestructured network as an analysis result of the anatomical treestructure analysis.
 20. The computer readable medium of claim 19,wherein the set of positions include the at least one bifurcation pointand at least one point in each branch.